In [19]:
__nbid__ = '0044'
__author__ = 'Brian Merino <brian.merino@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd; version datestamp of this notebook
__keywords__ = ['niri','gemini','supernova','dragons']

Gemini NIRI supernova reduction using DRAGONS Python API¶

Public archival data from niriimg_tutorial - GN-2015B-Q-31 (SN2014J)¶

adapted from https://dragons.readthedocs.io/projects/niriimg-drtutorial/en/v3.1.0/index.html¶

Table of contents¶

  • Goals
  • Summary
  • Disclaimers and attribution
  • Imports and setup
  • About the dataset
  • Downloading data for reduction
  • Set up the DRAGONS logger
  • Create File Lists
  • Create Master Dark
  • Bad Pixel Mask
  • Create Master Flat Field
  • Standard Star
  • Reduce Science Images
  • Display stacked final image
  • Clean-up (optional)

Goals¶

Showcase how to reduce NIRI imaging data using the Gemini DRAGONS package on the Data Lab science platform, using a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing flats, darks, a bad pixel mask, and science frames, and creating a single combined stacked image.

Summary¶

DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It can currently be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, as well as spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals, and tutorials about the use of DRAGONS.

The DRAGONS kernel has been made available in the Data Lab environment, allowing users to access the routines without being dependent on installing the software on their local machines.

In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to reduce example Gemini North NIRI H-band imaging data fully. This notebook will not present all of the details of the many options available to adjust or optimize the DRAGONS NIRI data reduction process; instead, it will just show one example of a standard reduction of a NIRI imaging dataset.

The data used in this notebook example is NIRI H-band imaging from the Gemini archive of the Supernova SN2014J from the Gemini North program "SN2014J at very late phases", PI: Marten van Kerkwijk, program ID GN-2015B-Q-31.

Disclaimer & attribution¶

Disclaimers¶

Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.

Acknowledgments¶

If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:

This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:

This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

In either case please cite the following papers:

  • Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445

  • Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411

If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:

  • Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097

If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}

And if you are using SPARCL, please also add \software{SPARCL} and cite:

  • Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024 https://doi.org/10.48550/arXiv.2401.05576

The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.

For this notebook specifically, please acknowledge:

  • DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L

  • DRAGONS open source software publication

Importing Python libraries¶

In [2]:
import warnings
import glob

from gempy.adlibrary import dataselect
from gempy.utils import logutils

from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce

from astropy.io import fits
from astropy.wcs import WCS
from astropy.utils.exceptions import AstropyWarning

import matplotlib.pyplot as plt
from matplotlib.colors import Normalize

warnings.simplefilter('ignore', category=AstropyWarning)

About the dataset¶

This is a NIRI imaging observation of an extended source, a galaxy showing as a dense field of stars. The observation sequence uses an offset to a nearby blank portion of the sky to monitor the sky levels since no area in the science observation is not "contaminated" by the galaxy.

The calibrations we use for this example include:

Darks for the science and sky offset frames. Flats, as a sequence of lamp-on and lamp-off exposures. Short darks to use with the flats to create a bad pixel mask. A set of standard star observations.

Observation Type File name(s) Purpose and Exposure (seconds)
Science N20160102S0270-274 on-target
Science N20160102S0275-279 on-sky
Science Darks N20160102S0423-432 20 sec, like science
Flats N20160102S0373-382 Lamp on
Flats N20160102S0363-372 Lamp off
Short Darks N20160103S0463-472
Standard Star N20160102S0295-299

Downloading the data¶

Downloading NIR images from the Gemini archive to the current working directory. This step only needs to be executed once.

If you run this notebook for the first time and need to download the dataset, set the variable "download=True". The notebook will not redownload the dataset if it is set to False. This will become particularly useful if you run the notebooks more than once.

In [3]:
%%bash 

# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/N20160102S0270.fits
http://archive.gemini.edu/file/N20160102S0271.fits
http://archive.gemini.edu/file/N20160102S0272.fits
http://archive.gemini.edu/file/N20160102S0273.fits
http://archive.gemini.edu/file/N20160102S0274.fits
http://archive.gemini.edu/file/N20160102S0275.fits
http://archive.gemini.edu/file/N20160102S0276.fits
http://archive.gemini.edu/file/N20160102S0277.fits
http://archive.gemini.edu/file/N20160102S0278.fits
http://archive.gemini.edu/file/N20160102S0279.fits
http://archive.gemini.edu/file/N20160102S0423.fits
http://archive.gemini.edu/file/N20160102S0424.fits
http://archive.gemini.edu/file/N20160102S0425.fits
http://archive.gemini.edu/file/N20160102S0426.fits
http://archive.gemini.edu/file/N20160102S0427.fits
http://archive.gemini.edu/file/N20160102S0428.fits
http://archive.gemini.edu/file/N20160102S0429.fits
http://archive.gemini.edu/file/N20160102S0430.fits
http://archive.gemini.edu/file/N20160102S0431.fits
http://archive.gemini.edu/file/N20160102S0432.fits
http://archive.gemini.edu/file/N20160102S0363.fits
http://archive.gemini.edu/file/N20160102S0364.fits
http://archive.gemini.edu/file/N20160102S0365.fits
http://archive.gemini.edu/file/N20160102S0366.fits
http://archive.gemini.edu/file/N20160102S0367.fits
http://archive.gemini.edu/file/N20160102S0368.fits
http://archive.gemini.edu/file/N20160102S0369.fits
http://archive.gemini.edu/file/N20160102S0370.fits
http://archive.gemini.edu/file/N20160102S0371.fits
http://archive.gemini.edu/file/N20160102S0372.fits
http://archive.gemini.edu/file/N20160102S0373.fits
http://archive.gemini.edu/file/N20160102S0374.fits
http://archive.gemini.edu/file/N20160102S0375.fits
http://archive.gemini.edu/file/N20160102S0376.fits
http://archive.gemini.edu/file/N20160102S0377.fits
http://archive.gemini.edu/file/N20160102S0378.fits
http://archive.gemini.edu/file/N20160102S0379.fits
http://archive.gemini.edu/file/N20160102S0380.fits
http://archive.gemini.edu/file/N20160102S0381.fits
http://archive.gemini.edu/file/N20160102S0382.fits
http://archive.gemini.edu/file/N20160103S0463.fits
http://archive.gemini.edu/file/N20160103S0464.fits
http://archive.gemini.edu/file/N20160103S0465.fits
http://archive.gemini.edu/file/N20160103S0466.fits
http://archive.gemini.edu/file/N20160103S0467.fits
http://archive.gemini.edu/file/N20160103S0468.fits
http://archive.gemini.edu/file/N20160103S0469.fits
http://archive.gemini.edu/file/N20160103S0470.fits
http://archive.gemini.edu/file/N20160103S0471.fits
http://archive.gemini.edu/file/N20160103S0472.fits
http://archive.gemini.edu/file/N20160102S0295.fits
http://archive.gemini.edu/file/N20160102S0296.fits
http://archive.gemini.edu/file/N20160102S0297.fits
http://archive.gemini.edu/file/N20160102S0298.fits
http://archive.gemini.edu/file/N20160102S0299.fits\
" > niri.list
In [4]:
%%bash

download="True"

if [ $download == "True" ]; then
    wget --no-check-certificate -N -q -i niri.list

else
    echo "Skipping download. To download the data set used in this notebook, set download=True."
fi
In [5]:
# # Check header of one raw science image
# tmp = fits.open("N20160102S0270.fits")
# tmp[0].header

Setting up the DRAGONS logger¶

DRAGONS comes with a local calibration manager that uses the same calibration association rules as the Gemini Observatory Archive. This allows reduce to make requests to a local light-weight database for matching processed calibrations when needed to reduce a dataset.

This tells the system where to put the calibration database. This database will keep track of the processed calibrations we will send to it.

In [6]:
logutils.config(file_name='niri_data_reduction.log')
caldb = cal_service.set_local_database()
caldb.init("w")

Create a list of all the FITS files in the directory

In [7]:
all_files = glob.glob('N2016*[0-9].fits')
all_files.sort()

Create file lists¶

This data set contains science and calibration frames. For some programs, it could have different observed targets and exposure times depending on how you organize your raw data.

The DRAGONS data reduction pipeline does not organize the data for you. You have to do it. DRAGONS provides tools to help you with that.

The first step is to create input file lists. The tool "dataselect" helps with that. It uses Astrodata tags and "descriptors" to select the files and send the filenames to a text file that can then be fed to "reduce". (See the Astrodata User Manual for information about Astrodata.)

Two lists for the darks

We have two sets of darks: one set for the science frames, the 20-second darks, and another for making the BPM, the 1-second darks. We will create two lists.

If you did not know the exposure times for the darks, you could have used a combination of "dataselect" to select all the darks (tag DARK) and feed that list to "showd" to show descriptor values, in this case, exposure_time. (See the descriptors page for a complete list.)

A list for the flats

The flats are a sequence of lamp-on and lamp-off exposures. We just send all of them to one list.

A list for the standard star

The standard stars at Gemini are normally taken as partner calibration.

A list for the science observations

The science frames are all the IMAGE non-FLAT frames that are also not the standard. Since flats are tagged FLAT and IMAGE, we must exclude the FLAT tag.

You can see the observation_class of all the data using "showd". Here, we will print the object name, too.

In [8]:
darks1s = dataselect.select_data(
    all_files, ['DARK'], [],
    dataselect.expr_parser('exposure_time==1'))

darks20s = dataselect.select_data(
    all_files, ['DARK'], [],
    dataselect.expr_parser('exposure_time==20'))

flats = dataselect.select_data(all_files, ['FLAT'])

stdstar = dataselect.select_data(
    all_files, [], [],
    dataselect.expr_parser('object=="FS 17"'))

target = dataselect.select_data(
    all_files, ['IMAGE'], ['FLAT'],
    dataselect.expr_parser('object!="FS 17"'))

Create master dark¶

We first create the master dark for the science target, then add it to the calibration database. The name of the output master dark, N20160102S0423_dark.fits, is written on the screen at the end of the process.

In [9]:
reduce_darks = Reduce()
reduce_darks.files.extend(darks20s)
reduce_darks.runr()

All submitted files appear valid:
N20160102S0423.fits ... N20160102S0432.fits, 10 files submitted.

Ginga not installed, use other viewer, or no viewer
No description has been provided for this image No description has been provided for this image

Bad pixel mask¶

The DRAGONS Gemini data reduction package, geminidr, comes with a static NIRI bad pixel mask (BPM) that gets automatically added to all the NIRI data as they gets processed. The user can also create a supplemental, fresher BPM from the flats and recent short darks. That new BPM is later fed to "reduce" as a user BPM to be combined with the static BPM. Using the static and fresh BPM from recent data leads to a better representation of the bad pixels. It is an optional but recommended step.

The flats and the short darks are the inputs.

The flats must be passed to the input list first to ensure that the recipe library associated with NIRI flats is selected. We will not use the default recipe, but rather the special recipe from that library called makeProcessedBPM.

The BPM produced is named N20160102S0373_bpm.fits.

In [10]:
reduce_bpm = Reduce()
reduce_bpm.files.extend(flats)
reduce_bpm.files.extend(darks1s)
reduce_bpm.recipename = 'makeProcessedBPM'
reduce_bpm.runr()

bpm = reduce_bpm.output_filenames[0]

Master flat field¶

A NIRI master flat is created from a series of lamp-on and lamp-off exposures. Each flavor is stacked, and then the lamp-off stack is subtracted from the lamp-on stack.

The master flat will be saved with the suffix _flats.fits

In [11]:
reduce_flats = Reduce()
reduce_flats.files.extend(flats)
reduce_flats.uparms = [('addDQ:user_bpm', bpm)]
reduce_flats.runr()

Standard star¶

The standard star is reduced more or less like the science target (next section), except that dark frames are not obtained for standard star observations. Therefore, the dark correction needs to be turned off.

The processed flat field we added to the local calibration database will be fetched automatically. The user BPM (optional but recommended) needs to be specified by the user.

The reduced standard star image will be saved with the suffix _image.fits.

NOTE: After running this cell, you may receive a warning regarding the Dark correction being turned off. You can ignore this warning as it will not affect the final product.

In [12]:
reduce_std = Reduce()
reduce_std.files.extend(stdstar)
reduce_std.uparms = [('addDQ:user_bpm', bpm)]
reduce_std.uparms.append(('darkCorrect:do_cal', 'skip'))
reduce_std.runr()

   WARNING - Dark correction has been turned off.

Reduce science images¶

The science target is an extended source. We need to turn off the scaling of the sky because the target fills the field of view and does not represent a reasonable sky background. If scaling is not turned off in this case, it results in an over-subtraction of the sky frame.

The sky frame comes from off-target sky observations. We feed the pipeline all the on-target and off-target frames. The software will split the on-target and the off-target appropriately.

The master dark and flat will be retrieved automatically from the local calibration database. Again, the user BPM needs to be specified on the command line.

The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the "SCI" extension, the variance is in the "VAR" extension, and the data quality plane (mask) is in the "DQ" extension.

The final reduced image will be saved with the suffix _image.fits.

In [13]:
reduce_target = Reduce()
reduce_target.files.extend(target)
reduce_target.uparms = [('addDQ:user_bpm', bpm)]
reduce_target.uparms.append(('skyCorrect:scale_sky', False))
reduce_target.runr()

Display the stacked image¶

In [14]:
image_file = "N20160102S0271_image.fits"
hdu_list = fits.open(image_file)
wcs = WCS(hdu_list[1].header)
hdu_list.info()

Filename: N20160102S0271_image.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     200   ()      
  1  SCI           1 ImageHDU        51   (1075, 1076)   float32   
  2  VAR           1 ImageHDU        51   (1075, 1076)   float32   
  3  DQ            1 ImageHDU        53   (1075, 1076)   int16 (rescales to uint16)   
  4  PROVENANCE    1 BinTableHDU     17   7R x 4C   [28A, 128A, 128A, 128A]   
  5  PROVHISTORY    1 BinTableHDU     17   17R x 4C   [128A, 428A, 28A, 28A]
In [15]:
image_data = fits.getdata(image_file, ext=1)
print(image_data.shape)

(1076, 1075)
In [16]:
plt.figure(figsize = (10,10))
plt.subplot(projection=wcs)
plt.imshow(image_data,cmap='bone',norm=Normalize(vmin=1, vmax=50000),origin='lower')
plt.xlabel('Right Ascension [hh:mm:ss]',fontsize=14,fontweight='bold')
plt.ylabel('Declination [degree]',fontsize=14,fontweight='bold')
plt.show()
No description has been provided for this image

Optional: remove raw data (uncomment lines before running)¶

In [18]:
# %%bash

# cp N20160102S0271_image.fits final.fits
# rm -r calibrations
# rm niri.list niri_data_reduction.log N2016*
# mv final.fits N20160102S0271_image.fits
In [ ]: